Music driven human dancing video synthesis

ABSTRACT

The present disclosure provides a method for generating a video of a body moving in synchronization with music by applying a first artificial neural network (ANN) to a sequence of samples of an audio waveform of the music to generate a first latent vector describing the waveform and a sequence of coordinates of points of body parts of the body, by applying a first stage of a second ANN to the sequence of coordinates to generate a second latent vector describing movement of the body, by applying a second stage of the second ANN to static images of a person in a plurality of different poses to generate a third latent vector describing an appearance of the person, and by applying a third stage of the second ANN to the first latent vector, the second latent vector, and the third latent vector to generate the video.

BACKGROUND 1. Technical Field

This disclosure relates to techniques for generating human dancingvideos from music, and more particularly to, techniques for generatinghuman dancing videos from music using neural networks.

2. Discussion of Related Art

Music and dancing are two related events that are often intertwined.Humans, especially professional dancers, are able to infer somereasonable dance movements when listening to music based on the rhythm,beat, and even genre.

The use of computerized programs to automatically generate reasonablehuman dancing videos from music may provide rich source materials andinspiration for choreographers. These programs may also facilitateediting of animations and movies, without people having to manually draweach frame for the animation or have a human actor practice a verydifficult dance to perform the dance professionally.

For example, a computerized program can generate predicted arms andfingers motion given piano or violin music, and apply 3D rendering tovisualize a predicted motion sequence, but the video generated would notappear to be human body movement that is realistic and smooth, norsynchronized well with music.

Thus, there are several challenges with regard to conventional systemsthat attempt to synthesize human body movement synchronized with music.

BRIEF SUMMARY

Embodiments of the present disclosure provide benefits and/or solve oneor more of the foregoing and other problems in the art with systems,non-transitory computer-readable media, and methods for creating videosincluding realistic and smooth human body movement in synchronizationwith music. In particular, in one or more embodiments, the disclosedsystem is implemented by a two-stage sequence-to-sequence framework thatgenerates human motion videos conditioned on input music. The two-stageframework can help to restrict the complexity of the conditional task ofgenerating a video of a target person dancing to given music byintroducing pose as an intermediate output.

The first stage (e.g., a music-to-pose neural network) includes twocomponents: (1) a waveform encoder and (2) a pose coordinates decoder.The music is represented using raw waveform samples. A first recurrentneural network (RNN) may be applied to encode the waveform intofeatures. Then, a second RNN may be used to decode the features intocoordinates of body part landmarks (pose).

The second stage (e.g., a pose-to-dance neural network) includes threecomponents: (1) a pose decoder, (2) an appearance encoder, and (3) avideo decoder. The pose encoder represents the pose sequence as a latentvector describing the body movement. The appearance decoder may be aconvolutional neural network (CNN) that takes a set of images of thetarget person in various poses as inputs and extracts a latent vectordescribing the target person's appearance. The video decoder may be anRNN that takes the concatenated latent features of waveform, pose, andappearance as inputs, and generates a sequence of video frames asoutput. In this way, the pose generation and appearance generation aredecoupled, making it easier to generate the same dance for differentpeople under the same music. This is particularly useful for generatingsynchronized dancing video for a group of people.

The video decoder is able to generate a target person's dancing video ina background scene, which is the same as the background scene in imagesof the target person in different poses that were initially fed to thesecond stage. If the images of the target person in the different poseshave a uniform background color, the output video will also have auniform background color.

Additional features and advantages of the above embodiments andadditional embodiments of the present disclosure are outlined in thedescription which follows, and in part will be obvious from thedescription, or may be learned by the practice of such exampleembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description describes one or more embodiments withadditionally specificity and detail through use of the accompanyingdrawings, briefly described below.

FIG. 1 illustrates a system including a human dancing video synthesizeraccording to an exemplary embodiment of the invention.

FIG. 2 illustrates an example of elements of the human dancing videosynthesizer according to an exemplary embodiment of the invention.

FIG. 3 illustrates an example of a sequence of coordinates that may beoutput by an element of the human dancing video synthesizer.

FIG. 4 illustrates an example of a first stage of the human dancingvideo synthesizer according to an exemplary embodiment of the invention.

FIG. 5A illustrates an example of an element of the human dancing videosynthesizer according to an exemplary embodiment of the invention.

FIG. 5B illustrates an example of using a mask of a body region formerging the human body with a background image.

FIG. 6 illustrates a method generating a video of a human being dancingaccording to exemplary embodiment of the invention.

FIG. 7 illustrates a method for performing a step of FIG. 6 according toexemplary embodiment of the invention.

FIG. 8 illustrates a schematic diagram of one embodiment of an exemplaryenvironment in which the music driven human dancing video synthesizercan operate.

FIG. 9 illustrates an exemplary computing device that may be used toperform one or more methods of the disclosure.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

One or more embodiments of the present disclosure include a music drivenhuman dancing video synthesizer that trains a music-to-pose neuralnetwork and a pose-to-dance neural network to generate a video of ahuman being dancing from input music and input images of the humanbeing. For example, in one or more embodiments, the music driven humandancing video synthesizer trains the music-to-pose neural network togenerate, from dancing videos including digital images and digitalaudio, feature vectors. The music driven human dancing video synthesizeralso trains the pose-to-dance neural network to generate a video basedon the feature vectors. The music driven human dancing video synthesizerthen utilizes the trained music-to-pose neural network and the trainedpose-to-dance neural network to generate a video of a human beingdancing in synchronization with music using input music and input imagesof the human being. The human dancing video synthesizer may also be usedto generate a video of multiple people dancing in synchronization withthe same music and with one another.

The music driven human dancing video synthesizer provides manyadvantages and benefits over conventional systems and methods. Forexample, the music driven human dancing video synthesizer's use of themusic-to-pose neural network combined with the pose-to-dance neuralnetwork provides better prediction of human body motion under ageneralized scenario, provides more subjective dancing motion, andprovides human body movements with a higher degree of freedom relativeto conventional systems. Specifically, application of the pose-to-danceneural network to actual images of a target person in numerous poses andsequences of coordinates of numerous body part landmark points output bythe music-to-pose neural network pre-trained on numerous videos ofpeople dancing to various different types of music allows thesynthesizer to generate realistic and smooth human body movements thatmatch and synchronize well with any input music.

Further, by use of the pose-to-dance neural network, the music drivenhuman dancing video synthesizer improves efficiency relative toconventional systems. For example, the pose-to-dance neural network iscapable of generating synthetic video of a person dancing to input musicwithout having to rely on an additional 3D renderer to visualize thepredicted motion. In particular, the pose-to-dance neural network'sapplication of a neural network to an aggregation of vectors thatdescribe a music waveform, poses of a generic body, and appearance of atarget person, enables the music driven video synthesizer to renderimages for a video without having to pass data to a separate external 3Drenderer.

Additional detail will now be provided regarding the music driven humandancing video synthesizer in relation to illustrative figures portrayingexemplary embodiments. In particular, FIG. 1 is a block diagramillustrating a system including the human dancing video synthesizer 100according to an exemplary embodiment of the inventive concept. Thesystem includes the human dancing video synthesizer 100, a waveformgenerator 110, and an image generator 180.

The waveform generator 110 processes received music 105 to generate araw audio waveform 120. In an embodiment, the waveform generator 110 hasa microphone that enables it to capture sound and an analog-to-digitalconverter to convert the sound into a raw audio waveform. For example,if a user is currently using a speaker to play a given selection ofmusic, the microphone can capture the sound and the analog-to-digitalconverter can periodically convert an analog voltage output by themicrophone that is representative of the sound to a digital value thatis representative of the amplitude of the sound. The waveform generator110 can then output a series of these digital values as the raw audiowaveform 120 to the music driven human dancing video synthesizer 100.

In another embodiment, the waveform generator 110 includes a userinterface that enables a user to select one of a plurality of differentpreviously stored music files, process the selected music file togenerate the digital values, and output the digital values as the rawwaveform 120 to the human dancing video synthesizer 100. The userinterface may also enable the user to search the Internet for music todownload so the raw audio waveform 120 can be extracted from thedownloaded music.

This direct use of the raw audio waveform 120 samples by the synthesizer100 prevents loss of information as opposed to prior techniques that usemel-frequency cepstral coefficients (MFCC) to represent the music 105.Since MFCC is a lossy representation of the music 105, the direct use ofthe raw audio waveform 120 by the synthesizer 100 ensures that all ormost information of the music 105 is retained.

In another embodiment, the waveform generator 110 is implemented withina client device (e.g., client device 802 a in FIG. 8) connected to themusic driven human dancing synthesizer 100 across a computer network(e.g., network 804 in FIG. 8). In this embodiment, the client deviceincludes a user interface that enables the user to select one of aplurality of different previously stored music files, process theselected music file to generate the digital values, and output thedigital values across the network 804 to the music driven human dancingvideo synthesizer 100. The client device may also include a microphoneto capture sound and an analog-to-digital converter to periodicallyconvert an analog voltage output by the microphone that isrepresentative of the sound to a digital value that is representative ofthe amplitude (or a peak) of the sound, and then output a series ofthese digital values as the raw audio waveform 120 across the network804 to the music driven human dancing video synthesizer 100.

The music driven human dancing video synthesizer 100 includes amusic-to-pose neural network (NN) 130 and a pose-to-dance NN 150. Themusic-to-pose NN 130 samples the raw audio waveform 120, applies a firstrecurrent neural network (RNN) to the samples to determine a waveformlatent vector L1 that describes the waveform 120, applies a second RNNto the waveform latent vector L1 to generate a sequence of coordinates(e.g., x and y coordinates) of points P (e.g., vertices) of body partsof a human body, and outputs the waveform latent vector L1 and thepoints P to the pose-to-dance NN 150. The pose-to-dance NN applies athird RNN to the waveform latent vector L1, the points P, and images 160of a person in a plurality of different poses to generate a video 170 ofthe person dancing. In an exemplary embodiment, each of the images 160is a still image of the person in a different pose. In an embodiment,the video 170 includes images of the person dancing and the music 105 sothat when the video is played, frames of images are sequentially playedwhile portions of the music are sequential played so that the imageframes are synchronized with the music portions.

The image generator 180 outputs the images 160 to the pose-to-dance NN150. In an exemplary embodiment, the image generator 180 includes adisplay device, a user interface that is presented on a display of thedisplay device, and a camera configured to capture the images 160. Forexample, the image generator 180 may control the user interface toprompt a user to stand (e.g., stand still) in a certain pose, and thenuse the camera to capture an image of the user after the user haspositioned themselves into the corresponding pose, change the certainpose, and then re-perform the prompting and capturing until a sufficientnumber of poses have been captured. In another embodiment, the imagegenerator 180 includes several cameras arranged around the user so theimages of the multiple poses can be captured at the same or atsubstantially the same time. In another exemplary embodiment, the imagegenerator 180 is located in a client device (e.g., 802 a) that outputsthe images across the computer network 804 to the pose-dance NN 150.

FIG. 2 illustrates examples of the music-pose NN 130 (1^(st) stage) andthe pose-dance NN 150 (2^(nd) stage) of FIG. 1 according to an exemplaryembodiment of the invention. The music-to-pose NN 130 includes awaveform encoder 131 and a pose decoder 132.

Prior to operating the waveform encoder 131 on a new waveform 120, thewaveform encoder 131 is trained. For example, the training may includefeeding the waveform encoder 131 a small chunk (e.g., a 1 secondduration chunk) of a training music waveform, and repeating this feedingusing a next chunk of the training music waveform until the entiretraining music waveform has been applied to the waveform encoder 131.The training may then be repeated on several other training musicwaveforms.

The trained waveform encoder 131 then samples the new waveform 120 at acertain sampling rate (e.g., 16 kHz sampling rate) for a certain periodof time (e.g., 1 second). The sampling results in a sequence of audiowaveform digits. For example, each digit may be an amplitude or a peakof the waveform 120 at the moment the sample was captured. In anexemplary embodiment of the invention, the sequence of digits is thanuniformly divided into small chunks and each chunk is fed into a longshort-term memory (LSTM) of the waveform encoder 131 recurrently toencode the samples into the waveform latent vector L1 that describes thewaveform 120. For example, if the sequence includes 10,240 digits, thesequence could be divided into 10 chunks, where each chunk is 1,024digits. In an exemplary embodiment, the digits in a chunk are normalizedbefore the chunk is fed to the LSTM of the waveform encoder 131. Forexample, the digits can be normalized to real values between −1 to 1.When a given chunk is fed to the LSTM of the waveform encoder 131, eachnode of a given layer of the LSTM takes both the current chunk of awaveform and the hidden state from a previous node. For example, theinformation learned from a previously encoded chunk is passed to a nextnode of the LSTM that operates on this information and the currentchunk. The sampling of the waveform 120 and the application of the LSTMcan then be repeated until the entire waveform 120 has been processed.

In an exemplary embodiment, the waveform encoder 131 and the posedecoder 132 are jointly trained. For example, the waveform encoder 131takes a first 1-second (or less) waveform sample of training music as afirst input to generate a first output, and then the pose decoder 132 istrained on the first output to generate first pose key points for thefirst 1-second of the training music, and then this is repeated untilall of the training music has been fed into the waveform encoder 131.The waveform encoder 131 learns first information when it is trainedusing the first waveform sample. Then, when the waveform encoder 131receives the second 1-second waveform sample of the training music, itgenerates a second output from the second waveform sample and the firstinformation. Similarly, the pose decoder 132 learns second informationwhen it is trained using the first output, and then when it receives thesecond output, it generates second pose key points for the second1-second of the training music using the second output and the secondinformation.

In an exemplary embodiment, training of the neural network (e.g., themusic-to-pose neural network 130) involves a first step of downloading avideo of a human being dancing from the Internet, extracting music fromthe downloaded video, extracting pose key points (body part landmarks)of the human dancing in the video, and labeling samples of the extractedmusic with the extracted pose key points to create music-pose key points(ground truth) pairs, and repeating the first step on several othervideos to create additional ground truth pairs. The training furtherincludes a second step of inputting music to the neural network topredict pose key points, comparing the predicted pose key points withthe ground truth pairs (extracted from the videos) to determineinformation indicating whether the network has learned correctinformation or not, and using the information to adjust the neuralnetwork.

In an exemplary embodiment, the pose decoder 132 includes an RNN thatdecodes the waveform latent vector L1 into the sequence of coordinatesof points P. The sequence of coordinates of points P may have anarbitrary length according to a duration of time of the waveform 120.For example, the length increases as the duration of waveform 120increases. In an exemplary embodiment, the points P are sequences ofcoordinates of body part landmarks. In an exemplary embodiment, as shownin FIG. 3, the landmark points include 14 points (e.g., center points)of a human being such as a head point PhN representing a point of ahead, a neck point PnN representing a point of a neck, a left shoulderpoint PlsN representing a point of a left shoulder, a right shoulderpoint PrsN representing a point of a right shoulder, a left elbow pointPleN representing a point of a left elbow, a right elbow point PreNrepresenting a point of a right elbow, a left wrist point PIwNrepresenting a point of a left wrist, a right wrist point PrwNrepresenting a point of a right wrist, a left hip point PIhNrepresenting a point of a left hip, a right hip point PrhN representinga point of a right hip, a left knee point PlkN representing a point of aleft knee, a right knee point PrkN representing a point of a right knee,a left ankle point PlaN representing a point of a left ankle, and aright ankle point PraN representing a point of a right ankle. Thesequence of points P include a time sequence of the head points (e.g.,Ph1, Ph2, . . . , PhN) representing motion of the head, a time sequenceof the neck points (Pn1, Pn2, . . . , PnN) representing motion of theneck, a time sequence of the left shoulder points (e.g., Pls1, Pls2, . .. , PlsN) representing motion of the left shoulder, a time sequence ofright shoulder points (Prs1, Prs2, . . . , PrsN) representing motion ofthe right shoulder, a time sequence of left elbow points (Ple1, Ple2, .. . , PleN) representing motion of the left elbow, a sequence of rightelbow points (Pre1, Pre2, . . . , PreN) representing motion of a rightelbow, a sequence of left wrist points (Plw1, Plw2, . . . , PIwN)representing motion of a left wrist, a sequence of right wrist points(Prw1, Prw2, . . . , PrwN) representing motion of the right wrist, asequence of left hip points (Plh1, Plh2, . . . , PIhN) representingmotion of the left hip, a sequence of right hip points (Prh1, Prh2, . .. , PrhN) representing motion of the right hip, a sequence of left kneepoints (Plk1, Plk2, . . . , PlkN) representing motion of the left knee,a sequence of right knee points (Prk1, Prk2, . . . , PrkN) representingmotion of the right knee, a sequence of left ankle points (Pla1, Pla2, .. . , PlaN), and a sequence of right ankle points (Pra1, Pra2, . . . ,PraN), where each sequence includes N points, where N is greater than 1.

In an exemplary embodiment, the pose decoder 132 includes an RNN thatoperates on the chunks of waveform samples to determine the landmarkpoints P. In an exemplary embodiment, the RNN of the pose decoder 132includes an LSTM, where each LSTM node of the LSTM predicts thecoordinates landmark points P of a human body for a current dancing posebased on the input music signal (e.g., a chunk of the waveform 120) aswell as the previously generated poses. For example, the second (middle)image in FIG. 3 including points P2 could be considered a currentdancing pose that was generated based on the input music 105 and aprevious generated dancing pose represented by points P1 in the first(left) image in FIG. 3. The first node of the LSTM of the pose decoder132 may be initialized with the last hidden state of the last node ofthe LSTM of the waveform encoder 131.

The pose-to-dance NN 150 includes a pose encoder 151, an appearanceencoder 152, an aggregator 153 (or a concatenator), and a video decoder154. For example, the pose encoder 151 may be referred to as a firststage of the pose-to-dance NN 150, the appearance encoder 152 may bereferred to as a second stage of the pose-to-dance NN 150, and theaggregator 153 and the video decoder 154 may be referred to as a thirdstage of the pose-to-dance NN 150.

In an exemplary embodiment, the pose encoder 151 includes an RNN thatoperates on coordinates of the landmark points P to generate a poselatent vector L2 that describes movement of the body. The pose latentvector L2 includes features of the landmark points P. In an exemplaryembodiment, different from the coordinates of the points P, the poselatent vector L2 is a fixed-dimension vector. The fixed-dimension vectormay contain abstracted motion information. For example, one dimension ofthe fixed-dimension vector may indicate the body is moving rightglobally, another dimension of the fixed-dimension vector may indicatethe speed of movement, and additional dimensions of the fixed-dimensionvector may indicate movement information about the body parts. Forexample, the additional dimensions may include a dimension thatindicates the left arm is rotating from a horizontal direction to avertical direction, a dimension that indicates the right arm is rotatingfrom the vertical direction to the horizontal direction, a dimensionthat indicates the head is moving from a vertical direction to a tilteddirection (e.g., 15 degrees rotated relative to vertical direction),etc.

In an exemplary embodiment, the appearance encoder 152 includes aconvolutional neural network (CNN) that operates on the images 160 of atarget person in various poses to extract an appearance latent vector L3that describes an appearance of the target person. The function of theappearance encoder 152 is to let the neural network (e.g., 150) knowwhich person (target person) is to be animated. In an exemplaryembodiment, the appearance encoder 152 applies multiple convolutionaloperations to the target person's picture until it has been encoded as afeature (L3). In an exemplary embodiment, the CNN is implemented usingResNet-50, which is a neural network that was trained on more than amillion images from the ImageNet database. The network is 50 layers deepand can classify images into 1000 object categories. In an embodiment,the object categories are used to pre-train the appearance encoder 152so it can encode the feature of the person correctly. In an embodiment,the ResNet-50 CNN is modified so that its last fully connected (FC)layer is replaced with another FC layer to output an M-dimensionalvector (e.g., M=1024) referred to as the appearance vector L3, and themodified CNN receives a concatenation of all the images 160 in a colorchannel as input.

The aggregator 153 concatenates all the vectors L1, L2, L3 to generateconcatenated data, and provides the concatenated data (e.g., thewaveform latent vector, the pose latent vector, and the appearancelatent vector) to the video decoder 154. In this way, the posegeneration is decoupled from the appearance generation, making it easierto generate the same dance for different people under the same music.This is particularly useful for generating synchronized dancing for agroup of people. The aggregator 153 may be implemented by a processor orsome other logic circuit. In an exemplary embodiment, the hidden statesfrom the LSTM of the pose encoder 151, the LSTM of the waveform encoder131, and the LSTM of the appearance decoder 152 are concatenated toinitialize the hidden state of the video decoder 154.

In an exemplary embodiment, the video decoder 154 includes an RNN thatis operated on the concatenated features of waveform, pose, andappearance to generate a sequence of video frames 170, which includevideo images of the target person dancing. In an exemplary embodiment,the video frames 170 further include the music 105 that was input to thewaveform generator 110 so that the target person is seen dancing to themusic when the video frames 170 are played. In an exemplary embodiment,the video decoder 154 stacks a frame generation architecture (e.g., agenerative model) with an LSTM layer to synthesize the depiction of theperson in corresponding dance poses.

FIG. 4 illustrates an example of the waveform encoder 131 and the posedecoder 132 according to an exemplary embodiment of the invention. In anexemplary embodiment, each of the waveform encoder 131 and the posedecoder 132 includes a 3-layer LSTM. For example, the 1^(st) layer ofthe LSTM of the waveform encoder 131 operates on a first waveform sampleX1 captured at a first time to generate features F1-1, the 2^(nd) layerof the LSTM of the waveform encoder 131 operates on features F1-1 togenerate features F1-2, and the 3^(rd) layer of the LSTM of the waveformencoder 131 operates on features F2-2 to generate features F1-3. The1^(st) layer of the LSTM of the waveform encoder 131 operates on asecond waveform sample X2 captured at a 1.0 second time and featuresF1-1 to generate features F2-1, the 2^(nd) layer of the LSTM of thewaveform encoder 131 operates on features F2-1 and F1-2 to generatefeatures F2-2, and the 3^(rd) layer of the LSTM of the waveform encoder131 operates on features F2-2 and F1-3 to generate features F2-3. The1^(st) layer of the LSTM of the waveform encoder 131 operates on an Nthwaveform sample XN captured at an Nth time and features F2-1 to generatefeatures FN2-1, the 2^(nd) layer of the LSTM of the waveform encoder 131operates on features F2-2 and FN-1 to generate features FN-2, and the3^(rd) layer of the LSTM of the waveform encoder 131 operates onfeatures F2-3 and FN-2.

The first layer of the LSTM of the pose decoder 132 operates on a hiddenstate H1 output by the first layer of the LSTM of the waveform encoder131 during a first time interval to generate first features, the secondlayer of the pose decoder 132 operates on the first features and ahidden state H2 output by the second layer of the LSTM of the waveformencoder 131 during the first time interval to generate second features,and the third layer LSTM of the pose decoder 132 operates on the secondfeatures and a hidden state H3 output by the third layer of the LSTM ofthe waveform encoder 131 to generate a first sequence of points P1 ofthe body parts. The pose decoder 132 may apply its LSTM to the hiddenstates output by the waveform encoder 131 during a second time intervalto generate a second sequence of points P2 of the body parts, and applyits LSTM to the hidden states output by the waveform encoder 131 duringan N-th time interval to generate an N-th sequence of points PN of thebody parts.

FIG. 5A illustrates an example of the video decoder 154 according to anexemplary embodiment of the inventive concept. As shown in FIG. 5A, thevideo decoder 154 includes an RNN 501 and a CNN 502. The waveform latentvector L1, the pose latent vector L2, and the appearance latent vectorL3 are concatenated, and a result of the concatenation is fed to an LSTMof the RNN 501. In an exemplary embodiment, the RNN 501 includes3-layers of LSTM units, which operates on the concatenation of theencoded feature from the pose encoder 152 (L2), the output of theappearance encoded 152 (L3), and the output of the waveform encoder 131(L1) to output encoded information that is fed to CNN 502 indicating thetarget person's appearance and what pose he should perform. The CNN 502will generate corresponding video frames 170 based on the encodedinformation. In an embodiment, the CNN 502 was previously trained usingdata like the encoded information.

In an exemplary embodiment, the pose latent vector L2 includescoordinates for several time intervals, which can be divided into groupscorresponding to each of the body parts. The numbers in each of thesegroups represent at least one of the initial central position of thecorresponding body part, its moving speed, and a 3D trajectory duringthe corresponding time interval.

In an exemplary embodiment, the waveform latent vector L1 represents ashort piece of music audio signal (e.g., 10 seconds). The numbers in thewaveform latent vector waveform latent vector L1 can be divided intodifferent groups, each corresponding to one of a plurality of differentfrequency bands. Different frequency bands may correspond to differentmusical instruments or different human singers. Each of these groups mayindicate at least one of a sound volume, a rhythm, or a tempo for acorresponding frequency band.

In an embodiment, the appearance latent vector L3 includes a pluralityof groups of numbers, where each group corresponds to one of the bodyparts. In an exemplary embodiment, the numbers in each group representat least one of the color, lighting, texture, and size of thecorresponding body part.

The LSTM of the RNN 501 performs an operation on the concatenated data,to generate an output that is fed to the CNN 502. The CNN 502 operateson the output to generate the video 170. The video encoder 154 knows thestart and end of the three vectors L1-L3 within the concatenatedtraining data during training. Based on what the video encoder 154 haslearned during this training, it is able to infer the start and end ofthe three vectors L1-L3 within newly input concatenated data.

In an exemplary embodiment, the video decoder 154 is able to generate adancing video 170 of a person (e.g., a human body) dancing in abackground scene, which is the same as the background scene of theimages 160 of the person in the plurality of poses. If the images 160have a uniform background color, the output video 170 will also have auniform background color, which enables simpler background replacementin the post-processing. In an exemplary embodiment, the video decoder154 generates images of the human body separately from an image of thebackground scene (e.g., background image) using the output of theaggregator 153 and merges the images of the human body with thebackground image to generate frames of the video 170. In an exemplaryembodiment, as shown in FIG. 5B, this merging is performed using a mask550 of a body region to merge an image of the human body with thebackground image to form a final resulting image 560. A first portion ofthe mask 550 resembling a human being may overwrite correspondingportions of the background image and a second portion of the mask 550surrounding the first portion may be transparent so that it does notoverwrite corresponding portions of the background image, when the mask550 is overlapped with the background image. When there are multipledancers, a similar process may be used to merge several human bodieswith one background image. The occlusion between people can be handledby a layer depth map.

In an exemplary embodiment, the LSTM for each of the encoders (e.g.,131, 151, and 152) and the decoders (e.g., 132 and 154) is a 3-layerLSTM. In an exemplary embodiment, the size of the hidden state of theLSTM of the waveform encoder 131, the LSTM of the pose decoder 132, andthe LSTM of the pose encoder 151 is set to 1024. In an exemplaryembodiment, the size of the hidden state of the video decoder 154 is3072.

In an exemplary embodiment, the music-to-pose NN 130 (1^(st) stage) andthe pose-to-dance NN 150 (2^(nd) stage) are trained separately. Forexample, the 1^(st) stage is trained first and then the 2^(nd) stage isappended to the first stage, and the whole system is trained together inan end-to-end manner. In an embodiment, the training includes randomlyinitializing weights of the system, and application of an Adamoptimization method to perform a gradient back-propagation. In anembodiment, when training the 1^(st) stage, an L1 loss (Least AbsoluteDeviation) is used between the ground truth pose coordinates andgenerated coordinates as an objective function. In an embodiment, whentraining the 2^(nd) stage, a combination of L1 pixel loss, Very DeepConvolutional Networks for Large-Scale Image Recognition (VGG)perceptual loss, pose consistency loss, and generative adversarial lossis used as a training objective to minimize the difference betweenpredicted and ground-truth video frames.

In an exemplary embodiment, the system (e.g. synthesizer 100) ispre-trained on a number (e.g., hundreds, thousands, etc.) of dancingvideos with different dancers and various genres for music and dance.The dancing videos may be downloaded from videos that are posted online.In an embodiment, a human pose extractor is run to extract ground-truthposes for training the 1^(st) stage. The audio and video signals of eachdancing video are split into short clips of a short duration (e.g., afew seconds) as individual training samples, and these short clips aresampled into frames at a first sampling rate (e.g., 15 frames persecond). In the second stage, the images 160 of the person are sampledat a second sampling rate (e.g., 3-5 frames per second) that is lowerthan the first sampling rate.

FIG. 6 illustrates a method generating a video of a human being dancingaccording to exemplary embodiment of the invention. The method of FIG. 6includes training a first ANN to output coordinates of body landmarkpoints based on input music (step 600). The method of FIG. 6 furtherincludes combining a second ANN to receive an output of the first ANN,and training the combination to output images of a person dancing to themusic (step 610). The method of FIG. 6 further includes sampling anaudio waveform to generate waveform samples (step 620). The method ofFIG. 6 further includes applying the first ANN to the samples togenerate a first latent vector describing the audio waveform andcoordinates of the landmark points appropriate for the waveform samples(step 630). The method then includes applying the second ANN to thecoordinates to generate a second latent vector describing movement, to aset of images of a target person in a plurality of different poses togenerate a third latent vector describing an appearance of the targetperson, and to all three latent vectors to generate the video (step640).

FIG. 7 illustrates a method that may be used to implement step 640 ofFIG. 6 according to an exemplary embodiment of the invention. The methodof FIG. 7 includes applying a first RNN to the coordinates to generate asecond latent vector describing movement (step 700). For example, thesecond latent vector may describe the movement of a stick figuregenerated from connecting the body landmark points of a single timetogether. The method of FIG. 7 further includes applying a first CNN tothe images of the target person in a plurality of different poses togenerate a third latent vector describing an appearance (step 710). Themethod of FIG. 7 includes concatenating all of the latent vectors (step720). For example, the first latent vector generated by step 630 may beconcatenated with the second and third latent vectors generated by step710 and 720, respectively. The method of FIG. 7 further includesapplying a result of the concatenating to a second RNN (step 730). Themethod of FIG. 7 then includes applying an output of the second RNN to asecond CNN to generate the video (step 740).

Referring back to FIG. 1, the synthesizer 100 may also be used togenerate a video of a group of people dancing in synchrony with musicand one another. For example, the images of the person 160 in multipleposes may be replaced with images of multiple people in different poses.In this embodiment, the appearance decoder 152 generates an appearancevector L3 for each of the different people, so that multiple appearancevectors L3 are created. For example, if there is a first person andsecond person, the video decoder 154 can operate on vectors L1, L2, andthe appearance vector L3 of the first person to generate first imageframes, operate on vectors L1, L2, and the appearance vector L3 of thesecond person to generate second image frames, and then perform amerging operation on the first and second frames to generate a video ofthe first person and the second person dancing in synchrony with thesame music and with one another.

FIG. 8 illustrates a schematic diagram of one embodiment of an exemplaryenvironment 800 in which the music driven human dancing videosynthesizer 100 can operate. In one or more embodiments, the exemplaryenvironment 800 includes one or more client devices 802 a, 802 b, . . .802 n, a network 804, and server(s) 806. The network 804 may be anysuitable network over which the computing devices can communicate.Example networks are discussed in more detail below with regard to FIG.9.

hi an exemplary embodiment, the waveform generator 110 is includedwithin one of the client devices 802 a, 802 b, . . . 802 n and the imagegenerator 180 is included within the same or another one of the clientdevices 802 a, 802 b, . . . 802 n, so that the waveform 120 and theimages 160 are sent across the network 804 to the synthesizer 100.

As illustrated in FIG. 8, the environment 800 may include client devices802 a-802 n. The client devices 802 a-802 n may comprise any computingdevice. For instance, in one or more embodiments, one or more of theclient devices 802 a-802 n comprise one or more computing devicesdescribed below in relation to FIG. 9.

In addition, the environment 800 may also include the server(s) 806. Theserver(s) 806 may generate, store, receive, and transmit any type ofdata, including the music-to-pose NN 130, the pose-to-dance NN 150, andtraining videos, training images, and training audio files for trainingthe synthesizer 100. For example, the server(s) 806 may transmit data toa client device, such as the client device 802 a. The server(s) 806 canalso transmit electronic messages between one or more users of theenvironment 800. In one example embodiment, the server(s) 806 comprise acontent server. The server(s) 806 can also comprise a communicationserver or a web-hosting server. Additional details regarding theserver(s) 806 will be discussed below with respect to FIG. 9.

As shown in FIG. 8, the server(s) can include a management system 808.The management system 808 can include an implementation of software,hardware, or both for managing one or more digital images, videos, oraudio files. For example, in one or more embodiments, the managementsystem 808 comprises software (such as ADOBE STOCK software) thatmaintains, manages, and/or provides a repository of digital images forutilization by a plurality of client devices. Similarly, in one or moreembodiments, the management system 808 comprises software that managesdigital images or audio files captured by a client device and stored atthe server(s) 806.

A server of the server(s) 806 may include a cloud-based service that isconfigured to execute on a host server, where the host server isconfigured to connect to a plurality of client devices 802 a, 802 b, . .. , 802 n over a computer network 804. A client application program maybe stored in one or more of the plurality of client devices 802 a, 802b, . . . , 802 n. In an exemplary embodiment, the client applicationprogram includes a user interface (e.g., a graphical user interface)configured to enable a user to select music and direct a camera of thecorresponding client device to capture images of the user in a pluralityof different poses. For example, the user interface could present theuser with a selectable list listing music files stored in a certaindirectory or the user interface could enable the user to search theInternet for the music. The client application program is configured tooutput an audio waveform 120 of the music 105 and the images 160 (e.g.,still images) across the computer network 804 to the cloud-based service(e.g., management system 808). The cloud-based service is configured toapply the audio waveform 120 to the music-to-pose neural network 130 togenerate the first latent vector L1 describing the audio waveform andthe sequence of coordinates of points P of body parts of the user, andapply the first latent vector L1, the sequence of coordinates, and theimages 160 (e.g., still images of a person in several different poses)to the pose-to-dance neural network 150 to generate the video 170 of theperson moving or dancing in synchronization with the music 105. Thecloud-based service can then output the video 170 across the computernetwork 804 to the client device (e.g., 802 a) from which it receivedthe audio waveform 120 and the images 160. A user of the client devicecan then use a media player of the client device to play the video 170.The cloud-based service may also stream the video 170 across thecomputer network 804 to the client device so the user can use the mediaplayer to play video data received in the stream. The stream may beoutput across the network 804 directly to the media player.

In an alternate embodiment, the cloud-based service is configured tooutput or stream the video 170 to a client device (e.g., 802 b) thatdiffers from the one it received the audio waveform 120 and the images160. In alternate embodiment, the music 105 or the audio waveform 120originates on the server 806 or is retrieved by the server 806, and auser of the client device uses a client application of the client deviceto select the music 105 or audio waveform 120 from the server 806 or todirect the server 806 to download the same, for processing bymusic-to-pose neural network 130.

As illustrated, in one or more embodiments, the server(s) 806 (and/orthe management system 808) can include all, or a portion of, thesynthesizer 100. In particular, the synthesizer 100 can comprise anapplication running on the server(s) 806 (as part of the managementsystem 808) or a portion of a software application that can bedownloaded from the server(s) 806. For example, the synthesizer 100 caninclude a web hosting application that allows the client devices 802a-802 n to interact with content hosted at the server(s) 806. Toillustrate, in one or more embodiments of the exemplary environment 800,one or more client devices 802 a-802 n can access a webpage supported bythe server(s) 806. In particular, the client device 802 a can run anapplication to allow a user to access, view, and/or interact with awebpage or website hosted at the server(s) 806.

Although FIG. 8 illustrates a particular arrangement of the clientdevices 802 a-802 n, the network 804, and the server(s) 806, variousadditional arrangements are possible. For example, while FIG. 8illustrates multiple separate client devices 802 a-802 n communicatingwith the server(s) 806 via the network 804, in one or more embodiments asingle client device may communicate directly with the server(s) 806,bypassing the network 804.

Similarly, although the environment 800 of FIG. 8 is depicted as havingvarious components, the environment 800 may have additional oralternative components. For example, the synthesizer 100 can beimplemented on a single computing device. In particular, the synthesizer100 may be implemented in whole by the client device 802 a orimplemented in whole by the server(s) 806. Alternatively, thesynthesizer 100 may be implemented across multiple devices or components(e.g., utilizing the client devices 802 a-802 n and the server(s) 806).

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 9 illustrates, in block diagram form, an exemplary computing device900 that may be configured to perform one or more of the processesdescribed above. One will appreciate that the music driven human dancingvideo synthesizer 100 can comprise implementations of the computingdevice 900. As shown by FIG. 9, the computing device can comprise aprocessor 902, memory 904, a storage device 906, an I/O interface 908,and a communication interface 910. In certain embodiments, the computingdevice 900 can include fewer or more components than those shown in FIG.9. Components of computing device 900 shown in FIG. 9 will now bedescribed in additional detail.

In particular embodiments, processor(s) 902 includes hardware forexecuting instructions, such as those making up a computer program. Asan example, and not by way of limitation, to execute instructions,processor(s) 902 may retrieve (or fetch) the instructions from aninternal register, an internal cache, memory 904, or a storage device906 and decode and execute them.

The computing device 900 includes memory 904, which is coupled to theprocessor(s) 902. The memory 904 may be used for storing data, metadata,and programs for execution by the processor(s). The memory 904 mayinclude one or more of volatile and non-volatile memories, such asRandom Access Memory (“RAM”), Read Only Memory (“ROM”), a solid-statedisk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of datastorage. The memory 1104 may be internal or distributed memory.

The computing device 900 includes a storage device 906 includes storagefor storing data or instructions. As an example, and not by way oflimitation, storage device 906 can comprise a non-transitory storagemedium described above. The storage device 906 may include a hard diskdrive (HDD), flash memory, a Universal Serial Bus (USB) drive or acombination of these or other storage devices.

The computing device 900 also includes one or more input or output(“I/O”) devices/interfaces 908, which are provided to allow a user toprovide input to (such as user strokes), receive output from, andotherwise transfer data to and from the computing device 900. These I/Odevices/interfaces 908 may include a mouse, keypad or a keyboard, atouch screen, camera, optical scanner, network interface, modem, otherknown I/O devices or a combination of such I/O devices/interfaces 908.The touch screen may be activated with a stylus or a finger.

The I/O devices/interfaces 908 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, devices/interfaces 908 areconfigured to provide graphical data to a display for presentation to auser. The graphical data may be representative of one or more graphicaluser interfaces and/or any other graphical content as may serve aparticular implementation.

The computing device 900 can further include a communication interface910. The communication interface 910 can include hardware, software, orboth. The communication interface 910 can provide one or more interfacesfor communication (such as, for example, packet-based communication)between the computing device and one or more other computing devices 900or one or more networks. As an example, and not by way of limitation,communication interface 910 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI. The computingdevice 1100 can further include a bus 912. The bus 912 can comprisehardware, software, or both that couples components of computing device900 to each other.

In the foregoing specification, the invention has been described withreference to specific example embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with less or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel with one another or inparallel with different instances of the same or similar steps/acts. Thescope of the invention is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes that come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

What is claimed is:
 1. A computer-implemented method for generating avideo of a body moving in synchronization with music, the methodcomprising: applying a first artificial neural network (ANN) to asequence of samples of an audio waveform of the music to generate afirst latent vector describing the audio waveform and a sequence ofcoordinates of points of body parts of the body; applying a first stageof a second ANN to the sequence of coordinates to generate a secondlatent vector describing movement of the body; applying a second stageof the second ANN to still images of a person in a plurality ofdifferent poses to generate a third latent vector describing anappearance of the person; and applying a third stage of the second ANNto the first latent vector, the second latent vector, and the thirdlatent vector to generate the video of the body moving insynchronization with the music.
 2. The method of claim 1, wherein theapplying of the first ANN comprises: applying a first recurrent neuralnetwork (RNN) to the sequence of the samples to generate the firstlatent vector; and applying a second RNN to the first latent vector togenerate the sequence of coordinates.
 3. The method of claim 1, whereinthe applying of the first stage comprises applying a first recurrentneural network (RNN) to the sequence of coordinates to generate thesecond latent vector, wherein applying of the second stage comprisesapplying a first convolutional neural network (CNN) to the still imagesto generate the third latent vector.
 4. The method of claim 3, whereinthe applying of the third stage comprises: concatenating all of thelatent vectors; applying a result of the concatenating to a second RNN;and applying an output of the second RNN to a second CNN to generate thevideo.
 5. The method of claim 1, wherein the video is generated toinclude a same background scene as a background scene in at least one ofthe still images.
 6. The method of claim 2, further comprising trainingthe second RNN prior to the applying of the second RNN, wherein thetraining comprises: extracting ground-truth poses from a plurality ofvideos including a person dancing; and training the second RNN using theextracted ground-truth poses.
 7. The method of claim 1, furthercomprising periodically sampling the music to generate the sequence ofsamples before applying the first ANN.
 8. The method of claim 7, whereineach sample includes a sequence of digits each representing an amplitudeof the audio waveform.
 9. The method of claim 7, wherein the periodicsampling comprises: determining a series of amplitudes of the audiowaveform over a plurality of time intervals; normalizing the amplitudesto values between −1 and 1; and generating the samples to respectivelyinclude a corresponding one of the values.
 10. The method of claim 2,wherein the samples are input to a first layer of a long short-termmemory (LSTM) of the first RNN and a first layer of a LSTM of the secondRNN operates on an output of the first layer of the LSTM of the firstRNN.
 11. The method of claim 10, wherein a first one of the samples isapplied to the first layer of the LSTM of the first RNN to generate ahidden state, and the hidden state and a second one of the samples isapplied to the first layer of the LSTM of the first RNN.
 12. The methodof claim 1, wherein the body parts include a head, a neck, left andright shoulders, left and right wrists, left and right hips, left andright knees, and left and right ankles.
 13. The method of claim 12,wherein the sequence of coordinates include at least two coordinates ofeach of the body parts at different times.
 14. The method of claim 1,where the first ANN is trained first and then the second ANN is appendedto the first ANN to generate a combination, and the combination istrained together in an end-to-end manner.
 15. A cloud-based system forgenerating a video of a user moving in synchronization with music, thesystem comprising: a cloud-based service configured to execute on a hostserver, the host server configured to connect to a plurality of clientdevices over a computer network; and a client application program storedin the plurality of client devices configured for execution by at leastone of the plurality of client devices, the client application programcomprising an interface configured to enable the user to select themusic and direct a camera of the corresponding client device to capturestill images of the user in a plurality of different poses, and theclient application program configured to output an audio waveform of themusic and the still images across the computer network to thecloud-based service, wherein the cloud-based service is configured toapply the audio waveform to a first neural network to generate a firstlatent vector describing the audio waveform and a sequence ofcoordinates of points of body parts, and apply the first latent vector,the sequence of coordinates, and the still images to a second neuralnetwork to generate the video of the body moving in synchronization withthe music, and wherein the cloud-based service is configured to outputthe video across the computer network to a media player of thecorresponding client device.
 16. The cloud-based system of claim 15,wherein the first neural network comprises a first artificial neuralnetwork (ANN) that is applied to samples of the audio waveform togenerate the first latent vector, and wherein the second neural networkcomprises a second ANN including first through third stages, wherein thefirst stage is applied to the sequence of coordinates to generate asecond latent vector describing movement of the user, wherein the secondstage is applied to the still images to generate a third latent vectordescribing an appearance of the user, and wherein the third stage isapplied to the first latent vector, the second latent vector, and thethird latent vector to generate the video.
 17. The cloud-based system ofclaim 16, wherein the first ANN includes a first recurrent neuralnetwork (RNN) that is applied to the samples to generate the firstlatent vector and a second RNN that is applied to the first latentvector to generate the sequence of coordinates, wherein the first stageincludes a third RNN that is applied to the sequence of coordinates togenerate the second latent vector, and wherein the second stage includesa convolutional neural network (CNN) that is applied to the still imagesto generate the third latent vector.
 18. The cloud-based system of claim17, wherein the third stage includes a fourth RNN that is applied to aconcatenation of all the latent vectors, and a second CNN that isapplied to an output of the fourth RNN to generate the video.
 19. Acomputer-implemented method for generating a video of a first person anda second person moving in synchronization with music and one another,the method comprising: applying an audio waveform of music to a firstneural network to generate a first latent vector describing the audiowaveform and a sequence of coordinates of points of body parts; applyingthe first latent vector, the sequence of coordinates, and still imagesof the first person in a plurality of different poses to a second neuralnetwork to generate first video frames of the first person; applying thefirst latent vector, the sequence of coordinates, and still images ofthe second person in the poses to the second neural network to generatesecond video frames of the second person; and merging the first andsecond video frames to generate the video of the first person and thesecond person moving in synchronization with the music and one another.20. The computer-implemented method of claim 19, wherein the firstneural network comprises a first artificial neural network (ANN) that isapplied to samples of the audio waveform to generate the first latentvector, wherein the second neural network comprises a second ANN that isapplied to the sequence of coordinates to generate a second latentvector describing movement of a body, to the first images to generate athird latent vector describing an appearance of the first person, and tothe first, second, and third latent vectors to generate the firstframes, and wherein the second ANN is applied to the second images togenerate a fourth latent vector describing an appearance of the secondperson, and to the first, second, and fourth latent vectors to generatethe second frames.